A building roof top in the aerial image may often not be aligned correctly with its building footprint. Such a misalignment may be problematic for large scale three-dimensional mapping and modeling in an urban area with multiple tall buildings. Additionally, buildings may not be correctly aligned with their respective footprints because of image degradation (e.g., noise, occlusion, and shadows). In conventional mapping and modeling systems, buildings are typically aligned one building at a time. Such conventional systems may not efficiently process building alignment on a large scale. Therefore, providing improved methods for refining building alignments is continuing effort.